GRAPLEr: A Distributed Collaborative Environment for Lake Ecosystem Modeling that Integrates Overlay Networks, High-throughput Computing, and Web Services

摘要

The GLEON Research And PRAGMA Lake Expedition -- GRAPLE -- is a collaborativeeffort between computer science and lake ecology researchers. It aims toimprove our understanding and predictive capacity of the threats to the waterquality of our freshwater resources, including climate change. This paperpresents GRAPLEr, a distributed computing system used to address the modelingneeds of GRAPLE researchers. GRAPLEr integrates and applies overlay virtualnetwork, high-throughput computing, and Web service technologies in a novelway. First, its user-level IP-over-P2P (IPOP) overlay network allows computeand storage resources distributed across independently-administeredinstitutions (including private and public clouds) to be aggregated into acommon virtual network, despite the presence of firewalls and network addresstranslators. Second, resources aggregated by the IPOP virtual network rununmodified high-throughput computing middleware (HTCondor) to enable largenumbers of model simulations to be executed concurrently across the distributedcomputing resources. Third, a Web service interface allows end users to submitjob requests to the system using client libraries that integrate with the Rstatistical computing environment. The paper presents the GRAPLEr architecture,describes its implementation and reports on its performance for batches ofGeneral Lake Model (GLM) simulations across three cloud infrastructures(University of Florida, CloudLab, and Microsoft Azure).